|
Purpose This paper provides information on background to the re-grading of WJEC’s GCSE English Language results in Wales. It supplements the oral statement made by the Minister for Education and Skills on the 25 September and the regulatory report that was published on 10 September in which the recommendation to re-grade was made. |
Introduction
1. The Education Act 1997 (as amended) gives the Welsh Ministers, amongst other things, functions in relation to qualifications. Since 2006, and the abolition of ACCAC, these functions have been exercised by the Welsh Ministers. The day to day exercise of these functions is undertaken by officials in what is now the Department for Education and Skills, on behalf of the Welsh Ministers
2. On 11 September 2012 I took the unprecedented step of using the powers given to the Welsh Ministers in the 1997 Act to direct WJEC to re-grade the GCSE English Language qualification for candidates in Wales. In so doing I acted upon the advice of my officials in their report GCSE English Language 2012 - an investigation into the outcomes for candidates in Wales which I had published on the previous day.
3. On 25 September 2012 I made an oral statement in plenary to update Assembly Members on the action taken and on 26 September the Chair of the Children and Young People Committee wrote to invite me to discuss the matter further with the Committee at a hearing on 8 November. I was asked, in particular, to provide further information on “discussions that took place between [my] officials, the WJEC and Ofqual regarding the changes to the [grade boundaries] for GCSE English Language and the options that were available to [me]”.
4. I provide further information on those discussions below.
Context
5. GCSEs and A levels have, historically, been three-country qualifications. That is to say, the same examinations and assessments have been taken by learners in Wales, England and Northern Ireland. There are currently five main awarding organisations (more commonly known as ‘exam boards’) offering GCSEs and A levels: WJEC, AQA, Edexcel, OCR and CCEA. Learners in all three countries have been able to take, and indeed have taken, GCSEs and A levels with any one of these boards – although CCEA, the exam board based in Northern Ireland, has this year announced that it will not continue to offer its GCSEs and A levels to learners in England.
6. Although WJEC is often referred to as “the Welsh Board”, more than half of its candidates for GCSEs and A levels overall are from England. This ratio varies from subject to subject, but for GCSE English Language over 70% of WJEC’s candidates this year were from England.
7. The three-country nature of these qualifications has always presented some challenges to the regulators in each of the three countries – especially in terms of the division of responsibilities. Decisions taken in one of the three countries have a direct impact on candidates in the other two countries. Therefore, while there has been some division of the regulatory workload, with Welsh Government officials taking a lead role in the regulation of qualifications offered by WJEC, it has been normal practice for regulatory officials to engage in three-way discussions with a view to reaching mutual agreement on any regulatory requirements, decisions or actions before these are communicated to exam boards. Typically, such communications have then been issued to exam boards on letters bearing the logos of all the three regulators – in order to promote consistency. Due to the relative size and capacity of Ofqual it has been the norm for these three-country letters to be issued by Ofqual on behalf of the three regulators.
8. The discussions between DfES officials and Ofqual and, to a lesser extent, between Welsh Government officials and WJEC in relation to GCSE English Language, took place in three main phases. These phases were:
i) prior to the determination and issue of the reporting guidelines to exam boards by Ofqual on 25 June 2012;
ii) following the awarding meeting on 27 July 2012 and prior to the publication of provisional GCSE results on 23 August 2012; and
iii) following the publication of provisional GCSE results and prior to the Direction to WJEC to re-grade the qualification on 11 September.
Discussions prior to the issue of reporting guidelines
9. Paragraphs 24 to 33 of the report I published on 10 September set out the background to the use of predictors in the determination of grade boundaries in GCSEs and A levels. In summary, the methodology was originally introduced to ensure that candidates taking a new or revised qualification were not unduly disadvantaged, or advantaged, by being the first cohort to take that qualification (this is referred to later in this paper as the “comparable outcomes principle”). The methodology is also designed to ensure that candidates of a similar ability are not disadvantaged, or advantaged, by taking their examinations with one exam board rather than another.
10. The predictor methodology was first employed across exam boards with the introduction of the current suite of A level qualifications – and sought, in that case, to maintain a stable relationship between a cohort’s performance at GCSE with its performance at A level. The introduction of this methodology for A levels was not unchallenged in Wales: Welsh Government officials and WJEC raised concerns about differences in the cohort and in the GCSEs that are taken in both countries. At the request of Welsh Government officials and WJEC, the three regulators commissioned NfER to investigate the reliability of the method – and, in the light of the findings of this research Welsh Government officials and WJEC both successfully challenged the tightness of the reporting tolerances. Nevertheless, and with some reservations, this methodology is broadly agreed to be applicable and relevant for A levels across England and Wales, while GCSE outcomes remain reliable and consistent across the two nations.
11. However, as set out in the report of 10 September, Welsh Government officials expressed continuous concerns about the application of a similar methodology whereby Key Stage 2 test results in England are used as the basis on which to form predictions for GCSE outcomes. Welsh Government officials were clear, from the outset, that such a methodology could not, and should not, be used in relation to candidates in Wales and, in 2011, they successfully argued with Ofqual that none of WJEC’s new GCSEs (in subjects such as Geography, History and Modern Foreign Languages) should have outcomes determined using Key Stage 2 results. While AQA, Edexcel and OCR determined their outcomes using this methodology in 2011, WJEC used, instead, a methodology designed to indicate stable outcomes between cohorts in centres with entries in two consecutive years.
12. Early in 2012, Ofqual officials alerted Welsh Government officials that they were minded, in the light of WJEC having substantial numbers of entries from candidates in England for some of their core subject GCSEs, to require WJEC to employ the use of Key Stage 2 predictors to determine the expected outcomes for WJEC’s specifications. Several discussions took place between Welsh Government and Ofqual officials with regard to the non-applicability of the predictors in Wales. However, Ofqual wrote to WJEC on 19 April, without the agreement of Welsh Government officials, asking them to report against Key Stage 2 predictions for all GCSEs where more than 500 candidates were from England. Welsh Government officials objected to this and continued to express the view that it was not appropriate for results for Welsh candidates to be determined on the basis of prior achievement by candidates in England. Nevertheless, when it became clear that a significant majority of WJEC’s candidature for GCSE English Language was from England, and that this represented the second largest awarding organisation entry for candidates from England, it was hard to justify continued resistance to the use of the methodology for that subject. It was therefore agreed that, for the English suite of qualifications only, WJEC would report their projected outcomes against the Key Stage 2-based predictions.
13. At no point in these early discussions was it apparent that the use of the KS2 predictor methodology would have any significant impact on the outcomes for candidates in Wales: it was only possible to see such an impact once candidates’ work had been marked and once the distribution of the marks was known (at the end of July 2012). There was also no indication that the comparable outcomes principle would not remain central to the implementation of any methodology for maintaining standards at GCSE level. It was also stated, in communications issued by Ofqual, that the use of predictors was ‘for reporting purposes only’. It had previously been the case that outcomes that fell outside the predicted outcomes had been accepted on the basis of appropriate and reasonable justifications for the difference between predicted and actual outcomes.
Discussions held between the Awarding meeting and the publication of results
14. Grade boundaries are normally determined at an awarding meeting by senior examiners at the exam board. Examiners review a range of candidates work to determine where a number of key grade boundaries should be set. In so doing, they are informed by a number of statistical indicators. In recent years, with the introduction of the predictor methodology, the weight given to these statistical indicators has increased.
15. In July 2012, WJEC’s Awarding meeting for GCSE English Language was delayed by some days due to concerns that had been raised, particularly by other exam boards, about difficulties with the statistical indicators. These difficulties are outlined in paragraphs 34 and 35 of the 10 September report. The Awarding meeting was finally held on Friday 27 July and was observed by a subject specialist, contracted by the Welsh Government as part of a programme of scrutiny for the qualification. That afternoon, prior to the meeting’s conclusion, the Chief Executive of WJEC, alerted a Welsh Government official to the likelihood that the outcomes at grade C and above for candidates in Wales seemed likely to be significantly down on the previous year’s outcomes. At that point in time it was not clear what the difference would be, but the official in turn alerted senior colleagues. On the same afternoon, that Welsh Government official spoke to a senior Ofqual official to:
· flag up the potential for WJEC’s outcomes being problematic,
· stress that the use of the predictor methodology for WJEC’s GCSE English Language qualification had been agreed on the basis of reporting purposes only. This would allow the regulators to review how WJEC’s outcomes compared with the predicted outcomes. If this process revealed a significant discrepancy between the predicted and actual outcomes, then under established procedures any gap would normally be closed in a staged approach; and
· state that it would not be appropriate for the outcomes for candidates in Wales to receive significantly lower outcomes due to the use of this methodology.
The Ofqual official did not raise any concerns with these points.
16. On Monday 30 July, the Chief Executive of WJEC, at the end of a previously scheduled meeting on other matters, alerted me to the likelihood that candidate outcomes in Wales for GCSE English Language were likely to be approximately 3.4% down on the previous year’s outcomes. I requested a full briefing from officials by the end of the week. I received this on Friday 3 August. Prior to submitting the briefing, my officials had attended a meeting on 31 July, hosted by Ofqual, in which concerns had been expressed by WJEC and other exam boards about the reliability of the predictor methodology on outcomes for GCSE English Language. Officials had also requested and received some further information from WJEC which raised concerns about the impact of the predictor methodology on outcomes for Wales and the Chief Executive of WJEC stated that “WJEC’s award for GCSE English Language provides outcomes for Wales that are considerably more severe than would be delivered under ‘comparable outcomes’ assumptions”.
17. The briefing that I received on 3 August provided a number of possible explanations for the likely fall in outcomes for candidates in Wales, which included a genuine fall in performance, particular difficulties in adjusting to the increased proportion of controlled assessment and problems with the reliability of the predictor methodology. Officials advised that the GCSE English Language outcomes would be the subject of a further meeting with Ofqual and the exam boards on Monday 6 August.
18. At that meeting on 6 August, hosted by Ofqual and attended by two Welsh Government officials, concerns were expressed by several exam boards about the effectiveness of the predictor model. Ofqual expressed concerns that WJEC’s provisional outcomes were “too generous” in comparison with the KS2 predictor model. Welsh Government officials argued that WJEC’s outcomes were too severe in terms of comparable outcomes for Wales and stressed that it appeared likely that candidates in Wales would suffer from the use of the predictor model, about which numerous concerns had been raised. They argued that consideration should be given to dis-applying the predictor model for GCSE English Language. The Chief Executive of Ofqual stated that she was not prepared to do this and decided that discussions about WJEC’s outcomes (and those of Edexcel which were similarly problematic) should continue outside the meeting.
19. It was clear from that meeting that agreeing a final set out of outcomes would be extremely challenging. As stated in paragraph 6 above, more than 70% of WJEC’s candidates for the qualification were in England, giving Ofqual a strong negotiating position. It should also be noted that it was viewed by all to be essential, at that time, to reach a single set of grade boundaries across the three nations.
20. On 7 August, discussions continued between Welsh Government officials and Ofqual. Welsh Government officials alerted the DfES Director General to the seriousness of the position. Ofqual were, by this time, insisting that WJEC should raise their grade boundaries for GCSE English Language as the provisional outcomes for GCSE English Language at grade C were 4.1 percentage points above the predicted outcome for candidates in England using the KS2 predictor methodology. Ofqual were clear that they would not accept WJEC’s awarding outcomes which already showed a significant fall for Welsh candidates.
21. On 8 August Ofqual wrote to WJEC asking them to review their grade boundaries and warning that if they did not do so, Ofqual would “consider it necessary to send… a notice of intention to issue a direction on this matter”.
22. On the morning of 9 August WJEC responded to Ofqual’s letter, copying their response to Welsh Government officials, providing data on four options for the grade boundaries. These options are set out in Annex 1. The first option was to retain the boundaries that were determined at the awarding meeting which already placed the outcomes at grade C and above, for candidates in Wales, 3.3 percentage points down on the previous year – but which put the results for candidates in England at 4.1 percentage points above the KS2 prediction.
23. The second option provided by WJEC put the Wales outcomes at C and above down by a further 0.6 percentage points from the awarding meeting outcomes and reduced the gap between outcomes in England and the KS2 predictions to 3.6 percentage points. It did, however, raise the grade A and above outcomes for Wales by 0.6 percentage points. In their response to Ofqual, WJEC indicated that, whilst arguing that the “at award” outcomes should stand, they felt that “if regulators’ collective view is that an adjustment should be made, we would suggest that [this option] is the one on which we would be able to reach agreement.”
24. The third option provided by WJEC put the Wales outcomes at C and above down from the awarding meeting outcomes by 1.7 percentage points (down 5 percentage points, from the outcomes at grade C in 2011) while reducing the gap for candidates in England between actual and predicted to 2.8 percentage points.
25. The fourth option put the Wales outcomes down from the awarding meeting outcomes by 2.7 percentage points (down 6 percentage points from the outcomes at grade C from 2011) and reduced the gap for candidates in England between actual and predicted to 2 percentage points.
26. Welsh Government officials discussed these options with Ofqual and argued strongly that the awarding outcomes, which were already damaging to candidates in Wales, should be retained. It was clear at this point that Ofqual would in no way accept the “at awarding” outcomes. At the same time, Ofqual wrote to Edexcel setting out Ofqual’s expectation that Edexcel should also bring their outcomes within tolerance of the Key Stage 2 predictors. It is my officials’ understanding that had Edexcel not complied with this, then Ofqual would have moved to issue a Direction to Edexcel. After lengthy discussions, in which Ofqual refused to accept the awarding outcomes, Welsh Government officials reluctantly agreed to accept the option referred to in paragraph 23 above, making it clear that this was not their preferred option but recognising the need for a common set of grade boundaries to be agreed at this stage.
27. On 10 August, Ofqual wrote to WJEC to ask them to implement these changes to the grade boundaries. Welsh Government officials insisted that this letter should not be sent on three-country letterhead but that the letter could indicate that discussions had taken place with Welsh Government regulatory officials and that agreement had been reached.
28. On 15 August I met with officials to discuss, primarily, the forthcoming A level results, but officials also briefed me on the events of the previous week in relation to GCSE English Language and explained how it had been necessary to agree a compromise.
29. I was briefed on the provisional GCSE results on 22 August prior to their issue on 23 August. It quickly became apparent that the results were problematic – not only in terms of the fall in outcomes in Wales but also in terms of the highly significant variations in performance between centres’ predicted and actual grades. On that day, I therefore asked regulatory officials to conduct a full investigation. In order to increase the objectivity of the investigation, the official who led on the investigation was one who had not been directly involved in the negotiations in early August.
Discussions held between the issuing of provisional results and the regrading of WJEC’s qualification in Wales
30. Shortly after I had initiated an investigation in Wales, Ofqual announced that they would also be conducting an investigation. A telephone conference was called between exam boards and regulators on Tuesday 28 August. Welsh Government officials leading on the investigation were in attendance and raised the dual concerns that the outcomes for candidates in Wales were down by more than a reasonable percentage on the basis of the comparable outcomes principle and that there was extreme variability between centres’ outcomes. Exam boards were asked to provide a large quantity of data to Ofqual. Welsh Government officials asked to receive copies of these and Ofqual agreed to forward the information on. In the event Ofqual did not forward any of the data. Whilst my officials believe this was an oversight, they did have to request data separately from the exam boards.
31. Telephone meetings between the regulators and exam boards, chaired by Ofqual, continued on a daily basis until Thursday 30 August and on Friday 31 August, Ofqual produced an interim report. It was the view of Welsh Government officials that the tone of the meetings indicated that Ofqual and the exam boards were agreeing a position of defence in order to confirm that they had taken appropriate action. There seemed to be little or no consideration of whether the results awarded had actually been fair to candidates. During this week, Welsh Government officials wrote to Ofqual on a number of occasions expressing their concerns and on 29 August stated that:
“In considering the final GCSE outcomes for Wales we have concluded that it is entirely unjustifiable and indefensible to let the WJEC outcomes stand as they are. We therefore intend to ask WJEC to adjust the outcomes of awarding for GCSE English Language using the 2011 Wales candidature as the basis for comparability and with a tolerance of 1% at grades C and A. This level of tolerance reflects the regulators’ level of tolerance for performance against predicted outcomes for a cohort of this size.”
32. After this email had been sent to Ofqual a short telephone meeting was held between Welsh Government officials and Ofqual. Ofqual stated that they did not agree with the Welsh Government’s position and would not be requesting a regrading.
33. On 29 August Welsh Government officials also wrote to WJEC to ask them to provide models for a potential regrading of the qualification. WJEC responded with three options. The first of these was to revert to the awarding outcomes which would have still meant a reduction at C grade of 3.3 percentage points. The second option would have improved this to a reduction of 2.6 percentage points. The third option would have resulted in a reduction of 1.5 percentage points.
34. DfES officials shared this data with Ofqual but again there was no indication of any willingness to explore such a regrading.
35. During the following week DfES officials prepared their investigatory report, requesting and compiling data from a number of sources. I received this report on the morning of Monday 10 September, approved its publication and accepted its recommendations. One of the key conclusions of the report was that the “outcomes for candidates in Wales for GCSE English Language at Grade C and above… are not secure or supported by any reasonable justification”. A central recommendation of the report was that WJEC’s qualification should be re-graded, preferably for all candidates, but in the event of Ofqual refusing to endorse the regrading, for candidates in Wales only.
36. Prior to publication the report was shared with WJEC, with Ofqual and with CCEA in Northern Ireland. A telephone conference was held, at Ofqual’s request, in which Ofqual requested that the publication of the report be delayed. In the interest of being able to take swift action to reduce the negative impact of the unfair grading on candidates in Wales and in the light of the fruitlessness of earlier discussions with Ofqual on the matter, this request was declined.
37. On 10 September a request was issued to WJEC asking them to issue, within 24 hours, an undertaking to re-grade GCSE English Language. When no such undertaking was received, I issued a Direction to WJEC to re-grade the qualification within one week. This regrading took place within the allotted timescale and as a consequence 2,386 candidates have now received the revised grades that I firmly believe they deserved.
Minister for Education and Skills
October 2012
Annex 1: Options presented to regulators by WJEC (paragraph 22 refers)
Option 1: Awarding Outcomes: KS2 Predictions for England and outcomes compared with Common Centre predictions and outcomes for Wales
|
|
Matched Entry |
A% |
C% |
F% |
|||
|
Predicted |
Actual |
Predicted |
Actual |
Predicted |
Actual |
||
|
English |
27517 |
1.2 |
1.7 |
36.1 |
38.8 |
96.7 |
96.3 |
|
English Language |
65927 |
19 |
19.3 |
77.3 |
81.4 |
99.5 |
99.6 |
|
Combined - England |
93444 |
13.8 |
14.1 |
65.2 |
68.9 |
98.7 |
98.6 |
|
Wales |
30247 |
14.3 |
12.2 |
64.6 |
60.4 |
98.7 |
98 |
Option 2: KS2 Predictions for England and outcomes compared with Common Centre predictions and outcomes for Wales
|
|
Matched Entry |
A% |
C% |
F% |
|||
|
Predicted |
Actual |
Predicted |
Actual |
Predicted |
Actual |
||
|
English |
27517 |
1.2 |
1.7 |
36.1 |
35.3 |
96.7 |
96.2 |
|
English Language |
65927 |
19 |
20 |
77.3 |
80.9 |
99.5 |
99.6 |
|
Combined |
93444 |
13.8 |
14.6 |
65.2 |
67.5 |
98.7 |
98.6 |
|
Wales |
30247 |
14.3 |
12.8 |
64.6 |
59.7 |
98.7 |
98.3 |
Option 3: KS2 Predictions for England and outcomes compared with Common Centre predictions and outcomes for Wales
|
|
Matched Entry |
A% |
C% |
F% |
|||
|
Predicted |
Actual |
Predicted |
Actual |
Predicted |
Actual |
||
|
English |
27517 |
1.2 |
1.7 |
36.1 |
35.3 |
96.7 |
96.2 |
|
English Language |
65927 |
19 |
18.9 |
77.3 |
80.1 |
99.5 |
99.6 |
|
Combined |
93444 |
13.8 |
13.8 |
65.2 |
66.9 |
98.7 |
98.6 |
|
Wales |
30247 |
14.3 |
11.9 |
64.6 |
58.6 |
98.7 |
98.3 |
Option 4: KS2 Predictions for England and outcomes compared with Common Centre predictions and outcomes for Wales
|
|
Matched Entry |
A% |
C% |
F% |
||||
|
Predicted |
Actual |
Predicted |
Actual |
Predicted |
Actual |
|||
|
English |
27517 |
1.2 |
1.6 |
36.1 |
35.3 |
96.7 |
96.2 |
|
|
English Language |
65927 |
19 |
18.9 |
77.3 |
79.3 |
99.5 |
99.6 |
|
|
Combined |
93444 |
13.8 |
13.8 |
65.2 |
66.3 |
98.7 |
98.6 |
|
|
Wales |
30247 |
14.3 |
11.9 |
64.6 |
57.6 |
98.7 |
98.2 |
|